System and method for measuring image quality

ABSTRACT

The present invention provides an improved system and method for measuring quality of both single and stereo video images. The embodiments of the present invention include frequency content measure for a single image or region-of-interest thereof and disparity measure for stereo images or region-of-interest thereof.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Provisional PatentApplication No. 61/020,554 filed Jan. 11, 2008, the entire disclosure ofwhich is incorporated herein by reference.

GOVERNMENT RIGHTS IN THIS INVENTION

This invention was made with U.S. government support under contractnumber 70NANB4H3044. The U.S. government has certain rights in thisinvention.

FIELD OF THE INVENTION

The invention relates generally to video image quality measures. Morespecifically, the invention relates to an improved system and method formeasuring stereo and single image quality.

BACKGROUND OF THE INVENTION

Over the past, several measures have been taken to detect impairments ofone or more stereo cameras of a stereo vision system. The forms ofimpairments can include camera obstruction of a partial or completefield of view blockage either by solid objects such as leaves on awindshield, environmental factors such as precipitation (water, snow orfog) or low-light conditions, problems with the optical system itselfsuch as poor camera focus, poor camera calibration, poor stereo camerarelative alignment or other unanticipated problems with the video.Additionally, low target contrast or texture (while not a cameraimpairment per se) can also cause poor system measurements when viewingthe video images. For example one of these impairments could cause acritical error in stereo measurements by altering the relativeorientation of the left and right camera, without benefit of acompensatory recalibration, which in turn would cause incorrectresulting depth computations, etc.

Collision detection systems are known in the art to compute stereoimages to detect potential threats in order to avoid collision or tomitigate its damage. The impairments could easily cause the collisionalgorithms to misidentify these incorrect measurements as potentialcollision threats, thus creating a false alarm, the effects of whichcould be drastic. Thus the presence of such impairments, onceidentified, should cause the system to temporarily disable itself forthe duration of the impairment, sometimes called a “failsafe” condition.This would be applicable also in less severe applications, which providefor much wider range of safety and convenience functions, for example,adaptive cruise control.

Stereo depth estimate accuracy can be computed precisely for a givenstereo algorithm on a stereo image data with known position and/orground-truth information. However, this ground-truth information may beunavailable or difficult to collect for real-world scenes, even incontrolled settings, and are certainly not available in the uncontrolledsettings of a deployed stereo or monocular imaging system. Moreover,such characterizations only measure the accuracy of a stereo algorithmunder ideal conditions, and ignore the effects of the kinds ofunanticipated impairments noted above. That is, a characterization of astereo algorithm's accuracy under ideal conditions does not predict andis not able to measure its robustness to various impairments found inuncontrolled real-world conditions.

Some algorithms may attempt to characterize specific impairments such asrain or fog in an operating imaging system using specificcharacteristics of the impairment itself (such as expected particle sizeand density), but may not generalize to other impairments such as hail,sleet or a sandstorm and therefore would not be able to reliably invokea needed failsafe condition. Thus the deployment of practical imagingsystems, particularly stereo imaging systems, has a need for a generalmeans to measure both monocular and stereo image quality.

SUMMARY OF THE INVENTION

The present invention provides a method for measuring image qualitycomprising receiving at least one input image in a region of interestand using an average adjacent pixel difference module for computing anadjacent pixel difference for each valid pixel in the region ofinterest, summing the at least some of the adjacent pixels' absolutedifferences and computing the average adjacent pixel difference withinthe region of interest.

In one embodiment of the present invention, the at least one input imageis an original image received from an image sensor and every region inthe pixel of interest is considered valid.

In another embodiment of the present invention, the at least one inputimage is a binarized disparity image formed from a stereo disparityimage. The stereo disparity image is computed from a stereo algorithmwhich explicitly labels each output pixel in the region as valid orinvalid.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates a block system diagram for measuring image quality inaccordance with an embodiment of the present invention.

FIG. 2 illustrates a flow diagram for measuring image quality inaccordance with an embodiment of the present invention.

FIG. 3 illustrates two successive pairs of stereo image frames alongwith their corresponding quality measures in accordance with anembodiment of the present invention.

FIG. 4 illustrates two successive pairs of stereo image frames alongwith their corresponding quality measures in accordance with anotherembodiment of the present invention.

FIG. 5 illustrate a pair of stereo image frames along with theircorresponding quality measures in accordance with an alternateembodiment of the present invention.

FIG. 6 illustrates a pair of stereo image frames along with theircorresponding quality measures in accordance with another alternateembodiment of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention describes various embodiments of image qualitymeasures. Two such measures include frequency Content measure formonocular image quality and disparity measure for stereo image quality.FIG. 1 illustrates a block diagram of a system 100 for measuring imagequality according to the two embodiments of the present invention. Thesystem 100 includes a first image sensor 102 a, preferably a left cameraand a second image sensor 102 b, preferably a right camera, forcapturing images in a region of interest (ROI). A first image 104 a,(preferably a left image) and a second image 104 b (preferably a rightimage) are captured from the first and second cameras 102 a and 102 brespectively. Note that even though a camera device is chosen as anexample for the image sensor, it is known to one skilled in the art,that any device that any device that functions as an image sensor may beused. Alternatively, the images captured from the sensor may bedownloaded and stored in any computer data storage device prior to beingprocessed. Such storage devices may include a disk, a memory card, adatabase in a computer, etc.

The images 104 a and 104 b are individually processed by an averageadjacent pixel difference (AAPD) modules 106 a and 106 b respectively toprovide the frequency content measure of the images as will be describedin greater detail below. Based on the frequency content measure, each ofthe left 104 a and the right images 104 b from each of the cameras 102 aand 102 b are then processed jointly by a stereo disparity module 108 toproduce a stereo disparity image as will be described in greater detailbelow. Alternatively, as illustrated in FIG. 1, each of the left and theright images 104 a and 104 b from both the left and the right cameras102 a and 102 b respectively are directly processed by the stereodisparity module 108 to provide the stereo disparity image 109 withoutmeasuring the frequency content of the images.

The stereo disparity image 109 is then converted into a binarized stereodisparity image 112 using a binarized disparity module 110. Thebinarized stereo disparity image 112 is further processed by an AAPDmodule 106 c to compute the disparity measure of the disparity image.Based on this disparity measure, the disparity image can either beforwarded for additional processing or ignored. If the disparity valuesare good, i.e. fall within a specific threshold, then the disparityimage 109 is further processed for various purposes such as objectdetection, collision detection etc. However, if the disparity values arepoor, i.e. fall outside the threshold value, then the disparity image109 is ignored and a new left and right images are obtained from each ofthe cameras 102 a and 102 b to repeat the process as will be describedin greater detail below.

Referring to FIG. 2, there is shown a flow diagram of a method formeasuring image quality according to the embodiments of the presentinvention. Initially, in step 202, an image in a region of interest(ROI) is captured from at least one of cameras 102 a or 102 b. In oneembodiment of the present invention, the image quality measure used formonocular image is frequency content measure. This measure includescomputing an average adjacent pixel difference (AAPD) in step 204 forone of the captured images 104 a and 104 b using either the AAPD Module106 a or AAPD module 106 b. The AAPD value for a given pixel in theimage is the sum of the absolute differences between its value and thevalue of its adjacent vertical and horizontal neighbors in the image.This prevents the difference values from being counted twice, andsuffices to Count only one of each of the vertical and horizontalneighbors. Then, the differences measured are summed up over the entireimage and further divided by a total number of valid pixels to measurethe frequency content in that image. In computing the AAPD value of thismonocular image, every pixel in the image is considered a valid pixel.So, high frequency content results from an image with a lot of finedetail, sharp edges and the difference between the pixel and itsadjacent neighbors will be relatively high, yielding a high AAPD value.On the other hand, if there is any type of impairment in which the imagehas a blur, poor focus, low local contrast, reduced texture etc., thefrequency content will be lower yielding a low AAPD value.

For example, FIG. 3 illustrates two successive pairs of stereo imageframes along with their corresponding quality measures according to anembodiment of the present invention. In the examples shown in FIG. 3,the image frames are taken both from the left camera and the rightcamera, however, the image quality frequency content measure displayedfor each frame from only from the right camera. In order to provide amore precise frequency content measure, this frame is preferably dividedinto thirds, a left third, a center third and a right third. Note thatthe division of the field-of-view is arbitrary, but dividing it intothirds is suitable for much on-road applications. So, the imagefrequency content is measured for the left side, the center and theright side of the image frames from the right camera. Image qualityvalues 92, 100, 100 shown in frame 1 are the image score values of thefrequency content converted in percentages for the left third, centerthird and the right third of the right camera image. In the sequence ofimage frames shown in FIG. 3, the right camera image is blocked from theleft side and the percentage image score value of the frequency contentleft third of the image lowers while the central third and the rightthird maintain their high spatial frequency content, i.e. good imagequality score. So, for example, the image frequency content values inimage frame 2 as values 75, 99 and 98 of FIG. 3 illustrate the imagedeterioration on the left side of the camera.

The frequency content measure for the AAPD measure described aboveapplies to images obtained from a single or a monocular camera,although, the monocular measure can be computed and compared with imagesfrom two or more cameras. The frequency content measure would be quitesimilar for two or more cameras looking at the same scene or view unlessthere is an obstruction in one or more cameras as discussed above, whichwould increase the difference in the frequency content of the twoimages. If this difference were measured to be high, an obstruction orimpairment of one of the cameras can be hypothesized.

In another embodiment of the present invention, the image qualitymeasure for binocular images is stereo disparity quality measure. Thestereo disparity quality measure includes producing a disparity image. Astereo disparity image is an image computed from images captured from atleast two or more stereo cameras which together produce a singledisparity image at a given resolution. Thus, unlike the frequencycontent measure, the stereo disparity quality measure is inherentlyapplicable to at least two camera images, most commonly the left and theright camera images of a stereo image pair. Returning back to the flowchart in FIG. 2, the steps 202 and 204 are repeated at least for thesecond camera out of the two cameras 102 a or 102 b. Upon receivingimage frequency content for the both images 104 a and 104 b, it isdetermined in step 206 if the values of the frequency content of theseimages are eligible for stereo disparity. The criteria for eligibilityof the frequency content values are determined empirically and areapplication specific. An example of the frequency content qualitymeasure for the monocular image is provided in table 1 below.

Monocular Image Frequency Content:

TABLE 1  80%-100% very good quality 60%-79% borderline quality 40%-59%seriously degraded 20%-39% obstructed  0%-19% obstructed

As illustrated in table 1 above, the image quality measure for monocularimage is return integer values between some lower bound and 100(%). Themonocular image quality measure ranges between 0-100. So, for the imageto be eligible for stereo disparity, the image frequency content valuesof each of the monocular images 104 a and 104 b must fall preferablywithin the range of 60% to 100% as provided in Table 1 above. The images104 a and 104 b having the frequency content values falling within therange of 0% to 59% are preferably considered to be ineligible for stereodisparity.

Referring back to the flow chart in FIG. 2, if is determined that thefrequency content values are not eligible, then steps 202 and 204 arerepeated for both the cameras 102 a and 102 b. However, if it isdetermined that the values are eligible, the stereo disparity image iscomputed in step 208. The stereo disparity image is computed by from theimages 104 a and 104 b captured by both cameras 102 a and 102 b using, aknown stereo disparity algorithm. Any stereo image disparity algorithmis suitable as input to the stereo disparity fragmentation measurementthat explicitly labels as invalid those points/pixels with insufficientcontrast or texture and labels as valid those points/pixels withsufficient contrast by algorithm's own definition for stereo matching.Note that the stereo image disparity algorithm determines the validityof the image based on a per pixel basis of validity, not a global or perregion basis of validity, as is done in the present invention. Oneexample of a suitable stereo output is that produced by the PyramidVision Technologies Acadia-I™, which computes stereo by finding the bestmatch using the Sum-of-Absolute-Differences (or SAD) matching measure.So, the stereo disparity image typically contains both valid and invaliddisparity pixels, with each pixel's status depending on some local imagequality criteria set by the given specific stereo disparity algorithm.

Alternatively, as shown in FIG. 2, the stereo disparity quality measurecan be obtained without measuring of the frequency content value. Thus,the stereo disparity image can be computed at step 208 directly from theoriginal images 104 a and 104 b captured at step 202 without firstdetermining the frequency content value at step 204. To compute stereodisparity quality measure, the disparity image is first converted into abinarized disparity image 112 at step 210 using the binarized convertedmodule 110. A value of 0 is given for all pixels with invalid regions(blank areas) and value of 1 for all pixels with valid regions (solidfigures).

Then at step 212, an AAPD is computed by the disparity AAPD module 106 cfor the binarized disparity image 112. Then, using the same AAPD oraverage-adjacent-pixel-difference function described previously (havingthe notable advantage of requiring only a single pass through thedisparity image pixels), the number of binary edge discontinuitiesbetween valid and invalid regions (both left-right and up-down) aresummed, then this sum is subtracted from the total number of validdisparity image pixels, then this result is divided by the total numberof valid disparity image pixels which yields the disparity qualityvalues. As discussed above, the stereo disparity algorithm defines thevalid and invalid disparity image pixels. A small number of largecohesive disparity regions will increase the disparity quality value,and a larger number of small, fragmented regions will decrease thedisparity quality value. Thus, the obstructed/degraded image frames havelower disparity quality values due to their many small disparity regionsand the unobstructed image frames will have higher disparity qualityvalues due to their lesser number of large disparity regions. Note thatthe sharing of the basic AAPD computational functionality can haveimplementation advantages; for example both the monocular and stereouses could share the same block of hardware such as a Field ProgrammableGate Array (FPGA) or Application Specific Integrated Circuit (ASCI).

Referring back to FIG. 2, after computing the stereo disparity qualityvalue at step 212, it is determined in step 214 if the values of thestereo disparity quality values are eligible for further processing. Ifit is determined that the values of the stereo disparity are noteligible for further processing, then the process is repeated beginningwith step 202. However, if the value is considered eligible for furtherprocessing, then the process for image quality ends and the images maybe forwarded for additional processing. The criteria for eligibility ofthe stereo disparity quality values are also determined empirically andare application specific. An example of the stereo disparity measure forthe stereo image is provided in table 2 below.

Stereo Disparity Measure:

TABLE 2  85%-100% very good quality 75%-84% acceptable quality 65%-74%borderline quality <0%-64% unacceptable

As illustrated in table 2 above, the image quality measure for stereodisparity image also returns integer values between some lower bound and100(%). The stereo disparity quality measure actually ranges between−299 and 100, but any values below 0 may be treated as 0 since thequality is already so poor. So, for the stereo disparity image 109 to beconsidered eligible for additional processing, the stereo disparitymeasure of the binarized stereo disparity image 112 must fall preferablywithin the range of 75% to 100% as provided in Table 2 above. Thebinarized stereo disparity image 112 having the stereo disparity valuesfalling within the range of any number below 0% to 74% are preferablyconsidered to be ineligible for additional processing.

For most real-world scenes without any camera impairments, the stereodisparity image fragmentation measurement is expected to be low, thatis, the stereo disparity quality is expected to be high. In other words,it is expected of the world to consist of mostly cohesive regions ofdisparity representing solid surfaces and objects. Conversely, cameraswith impairments of the kind described previously would experience manydisjoint regions of both valid disparity and invalid disparity pixels,and thus high disparity image fragmentation yielding a low stereodisparity quality, as the stereo algorithm struggles with theimpairments.

As discussed above, stereo disparity value is measured as the number ofvalid disparity pixels minus the number of valid/invalid pixeltransitions in the disparity image, divided by the number of validdisparity pixels. Thus a large roughly circular blob or region ofdisparity would have a low disparity image fragmentation, with a largenumber of valid disparity pixels and relatively small number ofdisparity region boundary valid/invalid pixel transitions. Conversely,multiple smaller regions of amorphous disparity would have a higherdisparity image fragmentation, with a high ratio of disparity regionboundary valid/invalid pixel transitions relative to its number of validdisparity pixels.

Referring back to the stereo image frame 1 of the FIG. 3, there is shownthe disparity image of the two original images having the stereodisparity values as 77, 88 and 86 of the left third, center third andthe right third of the image frame. Note again that the division of thefield-of-view is arbitrary, but dividing it into thirds is suitable formany on-road applications. As illustrated in the sequence of imageframes shown in FIG. 3, the right camera image is blocked from the leftside which further degrades the disparity value on the left third. So,for example, the disparity values 48, 84 and 84 in the stereo imageframe 2 illustrate the image deterioration on the left side of thecamera. This is clearly viewed on the stereo image frame 2 with theircorresponding disparity image in which the left side is turns into ablank area with few or no solid figures.

FIG. 4 illustrates another example a sequence of stereo image framesalong with their corresponding quality measures. In the examples shownin FIG. 4, the image frames are again taken both from the left cameraand the right camera, however, the image quality value, i.e. frequencycontent measure, for each frame from only the left camera is measuredwhile the disparity quality measure value is measured by combining theimages of both the left and the right cameras. Referring to stereo imageframe 1 of FIG. 4, there is shown image quality values for the monocularimage as 95, 99 and 95 of the left third, center third and the lightthird of the left image frame along with the corresponding left andright images. Also, illustrated in stereo image frame 1 of FIG. 4 is thecorresponding stereo disparity image which is fairly unobstructed andmainly consists of solid figure having the disparity values as 87, 89and 92 of the left third, center third and the right third of the image.The region on the disparity image enclosed in a rectangle corresponds tothe image of a car from the left camera and the right camera. Asillustrated in the sequence of image frames, specifically in image frame2 in FIG. 4, water is splashed onto the windshield of the car, however,the majority of the liquid falls only in front of the left camera withthe right camera receiving only a few drops. This degrades the frequencycontent value of the left third to 59 while the center and the rightthird remain as borderline quality as 76 and 76, as shown in stereoimage frame 2 of FIG. 4. This further degrades the disparity values onthe three sides of the camera images to 54, 60 and 57 and the disparityimage is now obstructed with the formerly big solid figures now brokendown into smaller regions with many blank areas, as viewed on the stereoimage frame 2 of FIG. 4. Thus, even though the right camera produces ahigh quality image, due to the blurriness in the left camera, it makesit very difficult to match the image received on the left camera withthe image received on the right camera, thus resulting in a very poorstereo disparity image.

The two examples as described and illustrated in FIGS. 3 and 4 aboveinclude case scenarios when both disparity value scores and thefrequency content value scores are both high and low at the same time.However, the present invention also includes cases in which either ofthe scores may be high while the other is low. For example, FIG. 5,illustrates images of the vehicle with effect of rain drops on thewindshield with no wipers. In this example, the individual camera AAPDvalues are high, due to the sharp image focus as illustrated by thefrequency content values as 98, 97 and 95 of the left third, center andright third of the left camera monocular image. However, since the raindrops visible on the windshield in front of one of the two stereocameras is entirely different from those of the other camera, the stereoalgorithm performs poorly and the disparity score is poor displayed as62, 70, and 61 of the left third, center and right third of the stereodisparity image in FIG. 5. In another case scenario, forwindshield-wiper obstruction example in FIG. 6, where the disparityquality score of the left side of the vehicle in the disparity image ishigh as displayed as 76, 35 and 86 but the frequency content scores ofthe left side of the vehicle in the disparity image are low as shown 53,50 and 92, which would correctly result in a rejection. This case canalso occur when the stereo calibration is very good and the stereoalgorithm can perform correctly in the presence of conditions such aspoor image focus.

Even though, the examples described above are related to the obstructiveview received by one or more cameras and poor focus settings, thedegradation may also occur due to camera misalignment. In that case, thestereo disparity measure will provide a poor score even though theindividual camera AAPD scores will be relatively good. This is becauseas discussed above, the AAPD computes scores based on the images from anindividual camera while the stereo measure compares the images based onthe images from at least two cameras. So, the AAPD is a good monocularimage quality measure, while the stereo disparity measure is focused onfragmentation of the computed disparity image.

Although, the present invention described above includes division of thefield-of-view into three ROIs for both image and disparity-based qualitymetrics, corresponding to the left, center, and right regions,additional regions could be included. More generally, any ROI ofinterest, for example one potentially containing a single object, may besubjected to one or more of these quality metrics. Alternatively, a moregeneral and dense image quality mask may be (generated by processingsmall regions around each pixel.

The techniques described are valuable because they provide a generalmeasure of stereo and single image quality, which is applicable againstunforeseen impairments as well as the examples described. Althoughvarious embodiments that incorporate the teachings of the presentinvention have been shown and described in detail herein, those skilledin the art can readily devise many other varied embodiments that stillincorporate these teachings without departing from the spirit and thescope of the invention.

1. A computer-implemented method for measuring image quality comprising:receiving at least one input image in a region of interest; and using anaverage adjacent pixel difference module for computing an adjacent pixeldifference for each valid pixel in the region of interest, summing theat least some of the adjacent pixels' absolute differences and computingthe average adjacent pixel difference within the region of interest. 2.The method of claim 1 wherein the at least one input image is anoriginal image received from an image sensor and every pixel in theregion of interest is considered valid.
 3. The method of claim 1 whereinthe at least one input image is a binarized disparity image.
 4. Themethod of claim 3 wherein said binarized disparity image is a disparityimage converted into said binarized disparity image by a binarizeddisparity module.
 5. The method of claim 4 wherein said disparity imageis computed from a stereo algorithm, said stereo algorithm explicitlylabels each output pixel in the region as valid or invalid.
 6. Themethod of claim 1 wherein the at least one input image in the region ofinterest comprise an entire image.
 7. The method of claim 1 wherein theat least one input image in the region of interest comprise a leftregion of interest, a central region of interest and a right region ofinterest.
 8. The method of claim 2 comprising computing a stereodisparity image if the average adjacent pixel difference of the originalimage is within a specific threshold.
 9. The method of claim 8comprising converting the stereo disparity image into a binarizeddisparity image.
 10. The method of claim 4 comprising forwarding thedisparity image for additional processing if the average adjacent pixeldifference of the binarized disparity image is within a specificthreshold.
 11. A computer-implemented system for measuring image qualitycomprising: an average adjacent pixel difference module for computing anadjacent pixel difference for each valid pixel in a region of interest,summing the at least some of the adjacent pixels' absolute differencesand computing the average adjacent pixel difference within the region ofinterest.
 12. The system Or claim 11 further comprising at least onesensor for capturing at least one image in a region of interest.
 13. Thesystem of claim 12 wherein the at least one image is the original imagereceived from the sensor and every pixel in the region of interest isconsidered valid.
 14. The system of claim 13 comprising a stereodisparity module for computing stereo disparity image from the originalimage using a stereo algorithm, said stereo algorithm explicitly labelseach output pixel in the region as valid or invalid.
 15. The system ofclaim 14 comprising a binarized disparity module for converting thestereo disparity image into a binarized disparity image.
 16. The systemof claim 14 wherein the at least one image is the binarized disparityimage.
 17. The system of claim 11 wherein the at least one image in theregion of interest comprise an entire image.
 18. The system of claim 11wherein the at least one image in the region of interest comprise a leftregion of interest, a central region of interest and a right region ofinterest.
 19. The system of claim 14 comprising computing a stereodisparity image if the average adjacent pixel difference of the originalimage is within a specific threshold.
 20. The system of claim 14comprising forwarding the stereo disparity image for additionalprocessing if the average adjacent pixel difference of the binarizeddisparity image is within a specific threshold.